Operation Clean Air: Clearing Up Misconceptions of Yelp’s Review Filter

Last week, the New York Attorney General’s “Operation Clean Turf” fined 19 companies a total of $350,000 for writing fake reviews on behalf of their clients. The case sets a laudable precedent not only for the future of local search, but for digital marketing more broadly.

While the amount of the fines is hardly Earth-shattering, the outcome of this operation should give pause to any SEO or reputation-management company considering quick-and-dirty, underhanded tactics to boost their clients’ rankings, “improve” their clients’ reputations, or launch negative attacks on competitors.

In the wake of this settlement, however, a wave of media coverage and a study by researchers at the Harvard Business School have clouded the reality of Yelp’s review filter—already poorly understood by typical business owners—even further. In this piece I hope to dispel four misconceptions that it would be easy to conclude from these recent publications.

1. “Most aggressive” review filter ≠ “most successful” review filter

Yelp representatives made little effort to contain their glee at being cited by the NYAG as having the “most aggressive filter” of well-known local review sites. In an interview with Fortune, Yelp’s corporate communications VP spun this statement by the NYAG as validation that his company’s filter was “presumably the most progressive and successful.”

As I stated in the same Fortune story, I agree 100% with the NYAG that Yelp’s filter is indeed the most aggressive. Unfortunately, this aggressiveness leads, in my experience, to a far higher percentage of false positives—i.e. legitimate reviews that end up being filtered—than the review filters on other sites.

Google, for example, has struggled for almost as long as Yelp to find the perfect balance between algorithmic aggression and giving users (and indirectly, business owners) the benefit of the doubt on “suspicious” reviews. Now that a Google+ account is required to leave a review of a business, I suspect that the corresponding search history and social data of these accounts give Google a huge leg up on Yelp in identifying truly fraudulent reviews.

I’m not necessarily saying that Google, TripAdvisor, Yahoo, or any other search engine presents the most representative review corpus, but it’s a pretty big stretch for Yelp to equate aggression with success.

2. “Filtered reviews” ≠ “fraudulent reviews”

To Yelp’s credit, even they admit that legitimate reviews are sometimes filtered out by their algorithm. But you sure wouldn’t know it by reading a recently published study by the Harvard Business School.

In a throwaway line that would be easy to miss, the authors state that they “focus on reviews that Yelp’s algorithmic indicator has identified as fraudulent. Using this proxy…” they go on to draw four—actually five—conclusions about “fraudulent” reviews:

1. Their star ratings tend to be more extreme than other reviews.
2. They tend to appear more often at restaurants with few reviews or negative reviews.
3. They tend to appear more often on independent restaurants rather than chains.
4. They tend to appear more in competitive markets.

5. “Fraudulent” 5-star reviews tend to appear more on claimed Yelp pages than unclaimed ones.

The authors attempt to use statistical equations to justify the foundation of their study, but the fundamental logic of their equations is flawed. I’m by no means a statistical wizard, but the authors suggest that readers like me scan filtered reviews to validate their assumption.

I would only highlight my friend Joanne Rollins’ Yelp page, and thousands of other business owners’ pages just like her, as qualitative evidence to rebut their logic. I don’t dispute that Yelp’s review filter is directionally accurate, but it’s crazy to assume it’s anywhere near foolproof to use it as a foundation for a study like this. It leads to self-fulfilling prophecy.

In fact, there are five very easy explanations of their conclusions that in no way lead you to believe that the overlap between filtered reviews and fraudulent reviews is even close.

1. Yelp uses star rating as part of its filtering algorithm. This is an interesting finding, but not applicable to “fraudulent” reviews.
2. Restaurants with few reviews or negative reviews are engaging in proactive reputation management by asking customers with positive experiences to review them. This is simply a best practice of online marketing. While it violates Yelp’s guidelines, by no means does it indicate that the reviews generated by these campaigns are fraudulent.
3. Independent restaurants tend to be much more engaged in online marketing than chains. Speaking from years of personal experience, chains have by-and-large been very slow to adopt local search marketing best practices, from search-friendly store locators to data management at local search engines to review campaigns. Independent small business owners simply tend to be more engaged in their digital success than corporate managers.
4. Restaurateurs in competitive markets tend to be much savvier about their digital marketing opportunities than those in less-competitive, typically rural markets.
5. Engaged restaurateurs are more likely to pursue proactive reputation management campaigns (see bullet-point number two).

While the HBS study highlights a number of interesting attributes of Yelp’s review filter, it’s simply impossible to draw the kinds of conclusions that the authors do about the truthfulness or fraudulence of filtered reviews.

3. “Filtered reviews” ≠ “useless reviews”

I consider my friend Joanne Rollins to be a fairly typical small business owner. She runs a small frame shop with the help of a couple of employees in a residential neighborhood of NW Portland. She’s not shy about sharing her ire with Yelp, not only around some of their shady sales practices, but especially about her customers’ reviews getting filtered.

Trying to explain some of the criteria that cause a review to be filtered simply takes too long, and Joanne is easily frustrated by the fact that a faceless computer algorithm is preventing testimonials from 13+ human beings from persuading future customers to patronize her business. On the customer side, they’re usually disappointed that they’ve wasted time writing comments that no one will ever see.

But all is not lost when a review is filtered! With permission from the customer, I encourage you to republish your filtered Yelp reviews on your own website. There’s no risk of running afoul of any duplicate content issues, since search engines cannot fill out the CAPTCHA forms required to see filtered reviews.

You as the business owner get the advantage of a few (likely) keyword-rich testimonials, and your customers get the satisfaction in knowing that hundreds of future customers will use their feedback in making a purchase decision. Marking these up in schema.org format would be the icing on the cake.

4. “Filtered reviews” ≠ “reviews lost forever”

A review once-filtered does not necessarily mean a review filtered-for-alltime. There are steps that I believe will make their review more likely to be promoted from the filter onto your actual business page:

• Complete their personal Yelp profile, including photo and bio information.
• Download the Yelp app to their mobile device and sign in.
• Connect their Facebook account to their Yelp profile.
• Make friends with at least a handful of other Yelpers.
• Review at least 8-10 other businesses besides yours.
• Leave at least one review with each star rating (i.e. 1-, 2-, 3-, 4-, 5-).

For those customers who are super-frustrated by Yelp’s filtering of their review or with whom you, as a business owner, have particularly a strong relationship, consider requesting that they undertake at least a couple of those tactics. I certainly don’t guarantee their success, but it’s worth a shot.

The reality of Yelp’s review filter

As the infographic above demonstrates, Yelp’s excitement over the citation from the NYAG as having the most aggressive filter underlines a fundamental business problem for the company that I’ve written about for years.

Yelp’s fortunes are tied to their success in selling business owners advertising. Yet these same business owners:

• don’t understand how the site works (at best)
• think that every Yelp salesperson is out to extort them (at worst)

Despite commendable efforts like their Small Business Advisory Council, Yelp clearly has a long way to go in educating these business owners. And they certainly have a long way to go with reining in rogue salespeople.

But the bigger issue is the consistent disconnect with their customers on the issue most important to their businesses–their guidelines for solicitation and display of reviews. Until they resolve that inherent conflict, I find it hard to see how they’ll grow their revenues to the levels that Wall Street clearly expects.